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ABSTRACT 

Current efforts and methodologies to evaluate 
vocational education programs are plagued by a number of problems. 
Included among these are the following: disagreement over the primary 
role and function of vocational education, too great an emphasis on 
summative and product evaluatiou as well as on labor market outcomes 
as indicators of program effectiveness, insufficient consideration of 
the educational and community setting, and overreliance on positivist 
evaluation inquiry. A new approach is needed to overcome these 
problems and deficiencies. One such approach is that of critical 
evaluation. A method based on the assumption that persons bring 
different perceptions and vested interests concerning an issue to the 
evaluation process, critical evaluation is concerned with both the 
process by which a decision is reached and the outcome or action 
itself. The critical evaluation approach directly addresses the 
issues raised by individuals in the local setting as being necessary 
for program change and relies upon quantitative and qualitative data 
to develop action plans and change strategies for program 
improvement. (MN) 
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FOREWORD 

The last two decades have seen the expenditure of consider- 
able resources on the evaluation of vocational education pro- 
grams. At the same time, there has laeen intensified criticism of 
the usefulness of evaluation findings, the criteria used in eval- 
uation efforts, and the methodology employed to evaluate voca- 
tional education programs. If program evaluation is to point the 
direction for program improvement in vocational education, then 
vocational education must be vdlling to reexamine program eval- 
uation and to identify and resolve the crucial problems limiting 
the effectiveness of evaluation. 

Research cn program effectiveness and school improvement 
shows the need to consider two crucial areas if desired changes 
are to occur: school culture and stakeholders' needs. Current 
evaluation approaches with their he. -y reliance on quantitative 
measures do not adequately address nese issues and are therefore 
of limited value. Critical evaluation, proposed in this publica- 
tion, overcomes those limitations, not by rejecting quantitative 
approaches, but by emphasizing the importance of stakeholder 
involvement; the need to consider the program's specific context? 
the need to analyze data in light of stakeholder values, beliefs, 
and attitudes; and the need to achieve consensus among 
stakeholders on proposed program improvement actions. 

It IS the intent of this publication to provide local and 
State vocational educators' with a brief overview and introduction 
to critical evaluation in vocational education. This is. not an 
implementation document, but rather an information source to 
broaden the awareness of the potential for positive change 
through the use of critical evaluation. Critical evaluation 
shows promise of enabling vocational educators to develop more 
practical and acceptable "plans for improving vocational education 
programs. 

The National Center expresses its appreciation to the 
following individuals who reviewed the manuscript or provided 
valuable input: Dr. Donna Coomer, Director of Planning and 
Evaluation, Private Industry Council, Menominee, Wisconsin; 
Dr. Cleo Cherryholmes, Professor of Political Science, Michigan 
State University, East Lansing, Michigan; Dr. John Klit, Manager 
of Program Approval and Evaluation, Illinois Department of Educa- 
tion, Springfield, Illinois; Dr. Gary Lloyd, State Specialist, 
Utah Department of Education, Salt Lake City, Utah; Dr. George 
Quarles, private consultant. New Rochelle, New York; and Dr. Ken- 
neth Sirotnik, Senior Research Associate, University of Califor- 
nia, Los Angeles, ' California . 

The National Center is indebted to the staff members who 
worked on the study that has resulted in this publication. The 
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study was conducted in the Evaluation and Policy Division, Dr. 
M. L. McCaslin, Associate Director. Dr. Floyd L. McKinney, Sen- 
ior Research Specialist, served aa project director. The project 
staff included Dr. Frank Pratzner, Senior Research Specialist, 
and Graduate Research Associates Joanne Farley, Alan Kohan, and 
Michael Smith. Priscilla Ciulla served as secretary for the 
project. Editorial review of the manuscript vas provided by 
Judy Balogh of the National Center's Field Services staff. 

Funding for the study was provided by the Office of Voca- 
tional and Adult Education, U.S. Department of Education. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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EXECUTIVE SUMMARY 



tionai program evaluation approaches are needed if voca- 

for^re ef?!^f • '"^^^ ^""^ ^^''^"^^ °^ ^^^^^ constituencies 

f^^^^I^^® effective programs. This publication reviews the 
limitations of current evaluation efforts and suggests critical 
evaluation as an approach that overcomes these dif icilncLsT 

brinn^S^fi^^^ evaluation is based on the assumption that persons 
to i5f?n%"- P^^°«P^i*^"« and vested interests about an issue 
situation i^ -^" process. This is important because change is 

oi?r fP^°^5^° involves moving the school culture from 
less effective to more effective ways of functioning. Critical 

rr'arriJ^d" at a'nd"th%"1ur''^ ^^^^^^^ ^^"^^^^ a'deJi^ion 

of rr^iT!!i ? outcome or action itself. Another tenet 

to deielon .h^i^o ^""^ "^^^ stakeholder involvement- 

reievlni^^rograj;"'""^ "^'"""'"^ support for appropriate and 

;.nd oi.rn^"''^ outflow from the findings on effective schools 
and the change process, critical evaluation directly addresses 
the issues raised by individuals in the local setting as being 
?A?^f^^^^. meaningful program change. Quantitative and quali- 
Iflt l: ^""^ ""^^^ ^° fJevelop action plans and change strate- 

nf !».f?^ program improvement. The benefits derived from the use 
of critical evaluation include stakeholder involvement, meanlnq- 

nn context-siSfecific recanmendations, and consensus 

on recommendations. . 
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CHAPTER 1 



THE HEED FOR NE\V DIRECTIONS 



Excellence in vocational education has become a major con- 
cern of vocational educators and the general public. The call 
for excellence has come at a time of rapid social and technolog- 
ical change coupled with fierce competition for public funds. 
These concerns have placed additional demands on vocational 
educators to improve programs. In order to improve programs, 
vocational educators need information about their programs— 
effort?? ' s>»oui<3 be provided through program evaluation 

Current E valuation Efforts in Vocational Education 

Much of the evaluation effort in vocational education has 
been driven by the requirements of Federal legislation. The 
Education Amendments of 1976, which mandated evaluation in terms 
of training-related placement and employer satisfaction with 
former vocational education students, have had the most impact on 
program evaluation. Additionally, States were required by the 
amendments of 1976 to review programs once every 5 years. The 
Carl D. Perkins Vocational Education Act of 1984 also designates 
specific measures -of effectiveness for vocational education pro- 
gram evaluation. The designated measures include the occupations 
for Which to be trained, which should reflect a realistic assess- 
ment of the labor market needs of the state, and the levels of 
skills to be achieved in particular occupations, which should 
reflect the hiring needs of employers. Additionally, the Act 
specifies that basic employment competencies to be used in per- 
formance outcomes, vihidh should also reflect the hiring needs of 
employers. The requirement for program evaluations by the State 
once each 5 years continues in the 1984 act. 

As long as States receive Federal funds that are subsequent- 
ly targeted to local programs, Federal regulations will have con- 
siderable impact on State and local evaluations. However, State 
and local vocational educators have nevertheless been relatively 
free to select the evaluation processes to be used in fulfilling 
these requirements. 



NOTE: For a more extensive treatment of the topics discussed in 
this publication the reader is referred to a companion publica- 
tion, J. Parley, F. McKinney, A. Kohan, M. Smith, and P. Pratz- 

Reconceptualiaation of Vocational Education Program Evalua - 
(Columbus: The Nationa; Center for Research in Vocational ' 
Education, The Ohio State University, 1985), 

s 
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states have generally complied with Federal requirements for 
prograun reviews by setting up review processes similar to the 
accreditation model of evaluation. These program reviews have 
been characterized by three sequential components: (1) a self- 
study (internal review) by the educational agency being evalu- 
ated? (2) an on-site visit by an external review team? and (3) a 
procedure for correcting deficiencies uncovered by the team. The 
internal review is most often' the responsibility of teachers and 
administrators and is usually seen as preparation for the on-site 
visit of the external review team. The on-site team visit is, in 
most states, a means of verifying the locally conducted self- 
study. External review teams usually esmphasize inputs and pro- 
cess variables. However, many local school personnel include 
placement rates and student achievement . as part of the informa- 
tion in the self-study report. 

Former student follow-up studies are frequently conducted by 
vocational educators. Follow-up studies conducted at the state 
and local levels vary greatly in their objectives, the kind of 
data collected, and the quality of the undertaking. In more re- 
cent years studies of employer satisfaction with former voca- 
tional education students have increased in use by local 
education agencies. 

Problems with Current Evaluation Efforts 

Program improvement is essentially a local initiative. Fed- 
eral and State goverranents may stimulate and support improvement 
efforts, but the commitment and activities essential to program 
improvement ranain centered at the local level, where programs 
are planned, conducted, and evaluated. 

If program evaluation is to contribute to program improve- . 
ment, it must provide useful information to vocational education 
decision makers and policymakers. However, current vocational 
education evaluation approaches have met with increasing criti- 
cism from those who believe they have not helped in local program 
improvement. Criticisms of current practices have often focused 
on the following five areas. 

Disagreement over the Primary Role and 
Function of Vocational Education 

The evaluation of vocational education is often based on a 
model that emphasizes the economic consequences of student's par- 
ticipation in vocational education. Federal legislation and reg- 
ulations have required attention to labor market outcomes? how- 
ever, this has frequently been inconsistent wxth the values and 
objectives of many local practitioners. Many contend that voca- 
tional education serves social and educational roles that are at 
least as important as its economic role. It is difficult, if not 
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impossible, to focus program evaluation activities When there is 
such sharp disagreement concerning the role and function of voca- 
tional education. 

Emphasis on Summative and Product Evaluations 

Most formal evaluations have focused on products or outcomes 
Of vocational education programs at the time of learner com- " 
pletion. Evaluations frequently use indicators such as job 
placeraent rates, employers' levels of satisfaction with former 
students, and former student tenure in jcbs. Summative and prod- 
^J^n^^Si'^fVv"^ important, but they seldom consider informa- 

tion about the activities and processes related to program 
outcones. Purthemore, they provide little direction in building 
an agenda for improving the vocational education program. Eval- 
uation results should lead to a better understanding of the rela- 
tionship of program inputs to outcomes. If program improvement 
IS the goal of evaluation, then vocational educators need an 
indication of which program processes to adjust to improve 
performance. 

Eva luative Emphasis on Labor Market Outcomes 

Program evaluation in vocational education has emphasized 
labor market outcomes with considerably lesa attention given to 
educational outcomes. The job placement rates and earnings of 
former students are frequently studied? other outcomes, such as 
interpersonal relations skills, problem- solving skills, economic 
literacy, career planning, and pursuit of additional education, 
have received minimal attention. 

Insufficient Consideration of the 
Educational and Community Setting 

The quality of learning is significantly affected by the 
total educational context (e.g., counseling processes, adminis- 
trative support) and community context (e.g., social mores, labor 
market structures) in which learning occurs. Program evaluations 
m vocational education have not adequately taken into account 
the processes and relationships within the school and the commu- 
nity that may contribute to program improvement. 

Overreliance on Positivist Evaluation Inquiry 

The dominant research and evaluation methodology in voca- 
tional education has been heavily influenced by the positivistic 
model of science. Proponents ansert that this model is the best 
way to acquire credible knowledge. Research and evaluation 
methodology based on this model reduces the human and social 
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aspects of vocational education to discrete, independent and de- 
pendent variables and is, thus, unable to capture *the complexity 
and holistic nature of vocational education programs. Moreover, 
the methodology relies on quantitative measurement and extensive, 
sophisticated statistical analysis. Thus, the variables selected 
for' evaluation and data analysis methods used are nearly always 
selected by "experts" Who are all too frequently far removed from 
the program being evaluated. 

Need for a New Approach 

The results of evaluation should increase the potential of 
policymakers and decision makers to improve vocational education 
programs. The typical approach to enhancing evaluation effec- 
tiveness in recent years has been to increase the sophistication 
of the methods, procedures, and techniques used. This strategy, 
which does not necessarily lead to more useful results, has 
alienated many individuals at the local school level. 

Policymakers tend to be enamored with quantitative data, 
but the irony is that they often make the most use of anecdotal 
data when engaged in policy-making. Vocational education is a 
complex and diverse array of programs, activities, and services. 
It exists in complex school and community environments. An 
evaluation approach is needed that provides a realistic picture 
of \/ocati6nal education and the environment in which it exists. 
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CHAPTER 2 
A CRITICAL EVALUATION FRAMEWORK 



In order to overcome the problems and deficiencies of the 
current progrzun evaluation efforts in vocational education a new 
framework is needed. The evaluation framework proposed in this 
publication is based in critical theory, it retains many of the 
beneficial aspects of current evaluation efforts and is designed 
to lead to improved vocational education programs. 



Theoretical Basis £or P rogram Evaluation Framework 

Critical theory is based on the assumption that persons 
bring to the evaluation process different perceptions of, and 
vesi:ed interests in a given issue. CSritical evaluation, empha- 
sizes the questioning of basic, common!^ held assumptions. This 
questioning, to the extent possible, should take place in an at- 
mosphere of free and nonthreatening communication among those who 
are involved in or affected by the educational enterprise, the 
stakeholders (e.g., teachers, administrators, students, counsel- 
ors, employers, parents, citizens). Unless stakeholder underly- 
ing assumptions are brought out in the open and recognized, 
meaningful consensus cannot be gained and plans for program im- 
provement cannot be realized. Critical theory is as concerned^ 
with the process by which a derision is reached as with the out- 
conje or action resulting from the decision. 

Program evaluation based on critical theory is concerned 
with (1) the understandings and motives resulting" in human 
action, (2) the way programs are altered as individuals develop 
better understandings of program processes and structures, 

(3) the role of evaluators as participants in the program 
improvement process rather than as disinterested observers, and 

(4) stakeholders engaging in a self-education process about the 
vocational education program and the environment in which it 
exists. In critical evaluation, the self-education process is 
seen as one that permits understandings and actions to develop, 
with the result being that programs change to become consistent 
with enlightened viewpoints. 

The Evaluation Framework 

Many people think of evaluation as a precise step-by-step 
process. While critical evaluation may proceed according to a 
logical order of activities, it is likely that many of the activ- 
ities will occur during the same time period. An action at one 
point in time may mean that an action taken previously needs to 



5 13 



be reconsidered and possibly altered. Figure 1 provides an il- 
lustration of critical evaluation in vocational education. To a 
great degree, the activities shown in figure 1 are interactive. 



Context for Critical Evaluation 

A context for generating evaluation knowledge about the 
vocational education program ts created when (V) agreement among 
stakeholders regarding shared viewpoints break down or are called 
into question, and (2) the communication or interaction among 
stakeholders ordinarily us^d to resolve questions is no longer ^ 
understood by the stakeholders (i.e., it is not shared communica- 
tion) . The four conditions necessary for shared oral or written 
communication are: 

o Statements must be comprehensible ztnd intelligible; that 

is, involved stakeholders must be able to understand each 

other. Misunderstandings must be clarified before 
further communication can occur. 

o Statements must be accurate or true. Statements should 
be congruent with all available and mutually recognized 
pertinent information about the problem. 

o Statements must be sincere in intent (e.g., there must be 
no hidden motives behind the communication). The indivi- 
duals engaged in conversation must be truthful and trust- 
ful of each other's intentions. 

o Communication must occur in a context where individuals 
feel free to express themselves without fear of reprisal. 

As shown in figure 1, the ongoing reflection and discourse 
are central to the conduct of critical evaluation. These are not 
methodological steps? rather, they pervade the critical evalua- 
tion process. Reflection is the process of identifying, making 
explicit, and critically examining all of assumptions, values, 
and beliefs underlying the knowledge and understandings that are 
accepted or taken- for-gr an ted by stakeholders. It is an ongoing, 
continuous process engaged in by stakeholders to validate period- 
ically existing beliefs and to identify distortions, changes, or 
mismatches that may have occurred over* time between accepted 
knowledge and understandings and the underlying norms or stand- 
ards. Engaging in reflection helps stakeholders discover why 
things are the way they are and develop insights about ways to 
improve programs. 

Discourse is the formal, lengthy, earnest., and intelligent 
discussion of a subject, either written or spoken. It has also 
been defined as the process or power of reasoning. In critical 
evaluation, the term is used in both of these senses. Discourse 
is a means of resolving questions about the accuracy and truth of 
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RBflectl¥e and 
Critical DIacourse 




FRAMtNG EVALUATION 
PROBLEMS 

• Understanding program 
history and culture 

• Identifying stalceholder's 
assumptions and 
presuppositions 



CONTEXT FOR 
GENERATING 
EVALUATIVE KNOWLEDGE 



Reflactl¥a and 
Critical Dlacouraa 



Reflactl¥e and Critical Discourse 



INTERPRETING 
DYNAMICS OF PROBLEMS^ 

• Clarifying distorted 
understandings 

• Constructing interpretive 
explanations 



Figure 1. Critical evaluation in vocational education 
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viewpoints or knowledge claims. Through discourse, stakeholders 
arrive at a consensus on statements of knowledge that are logi- 
cal, accurate, and rational. Like reflection, critical discourse 
must be an ongoing activity in a critical evaluation. 

In evaluating a vocational education program, it is impor- 
tant to understand how the program has evolved over time through 
the development of certain accepted values or norms. Through 
canmunicative action, individuals and groups that are members of 
a community have agreed on these values or norms. These norms 
have provided the context in vfliich certain community interests 
are achieved through the vocational education program. For ex- 
ample, in some communities, it is deemed important for the school 
to provide students with meaningful cooperative work experiences 
related to student career goals. The values placed on coopera- 
tive work experience are then an important part of ■ the contextual 
information essential for effective program evaluation. 

These norms vary not only from one community to another but 
also within communities. Ethnic groups, racial groups, age 
groups, and gender groups within a given community may be guided 
by radically different norms. The critical evaluation framework 
encourages those engaged in evaluation efforts to seek an under- 
standing of these differences and how they might affect the 
vocational education program. 



Framing Problems in Critica l Evaluation 

As indicated in figure 1, framing problems includes both the 
identification of problems to be addressed during the evaluation 
and the interpretation stakeholders have of these problems, based 
on their experiences. 

Particular attention must be paid to the reference points 
stakeholders use in interpreting vocational education programs. 
The viewpoints stakeholders hold are heavily influenced by their 
ideas and perceptions of vocational education and their relation- 
ship to it. These ideas and perceptions may be fairly consistent 
over tiiy or they may be revised as circumstances change. An ex- 
ample Of this change can be seen in how some stakeholders per- 
ceive the need for a school-based job placement office. When 
jobs are easy to find, the job placement office may not be deemed 
very important. However, in times of high unemployment, these 
same stakeholders may view the job placement office as being very 
important . 

Understanding the cultural setting and clarifying viewpoints 
are activities that proyide the backdrop for formulating evalua- 
tion problems or questions to be addressed. As stakeholders 
interact with each other, obstacles to the vocational education 
program will become evident, as will any discrepancies between 
intended and actual program developments. An example of this id 
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the discrepancy between the former Federal legislative emphasis 
on 30b placement and the emphasis of some programs on 
employability skills. For many local schools, this situation 
creates an evaluation problem relating to program goal 
clarification. Once such evaluation problems are formulated, 
they are not set aside but are analyzed throughout the evaluation 
process for possible modifications. 
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Interpreting Problem Dynamics in Critical Evaluat 

A major consideration in critical evaluation, as shown in 
figure 1, is clarifying understandings and constructing explan- 
ations. One of the major aims of critical, evaluation is to 
develop understandings of how and why the vocational education 
program has evolved into what it is. Only in this way can stake- 
holders clearly identify problem sources and constraints. One of 
the distinctive characteristics of critical evaluation is the 
participation of stakeholders in t^his interpretive process. Such 
participation is necessary because the activities and attendant 
beliefs, values, and meanings of individual stakeholders make up 
the vocational education program. That is, at least some stake- 
holders need to feel that they "own" the idea for change or inno- 
vation. Thus, program improvement can occur only through the 
conscious activities of these individuals and groups. 

Participation is also necessary in order to conduct dis- 
course examining the accuracy of stakeholder explanations about 
the vocational education program. Participation in discourse 
helps to identify possible misinterpretations on the part of 
stakeholders, such as mistaken readings of the situation, in- 
appropriate value positions, or claims based on self-serving 
interests. Explanations based on misunderstanding are distorted? 
thus, understanding must be sought through discourse a.iiong rele- 
vant stakeholder groups. 

This process of achieving understandings is similar to that 
of deciphering a code. Oie is frequently confronted with an ar- 
ray of complex information. One may need to use va?-ious method- 
ologies for obtaining information about the probl an being evalu- 
ated. State-of-the-art research findings may also be brought to 
bear in order to make sense of the problem. Interpreting the 
problem involves data gathering, social and historical analysis, 
and stakeholder reflection and discourse. Interpreting the prob- 
lem gets at questions lying at the heart of critical evaluation: 
Why? How did it come about? In whose interests? How does it 
fit into the larger educational, social, and political contexts? 

Finally, it should be noted that the critical evaluation 
process cannot be expected to yield a single explanation that > 
will be valid for all time and in all contexts. Rather, it can 
be expected to yield a valid explanation of current vocational 
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education reality that will clarify sources of current problems 
and suggest opportunities and strategies for modifying programs. 
With the passage of time, this interpretive explanation must be 
subjected to reanalysis, critiqued, and reconstructed. 



Overcoming Problems in Critical Evaluation 

As shovm in figure 1, once the vocational education reality 
is clarified, obstacles to program improvement efforts must be 
identified. Obstacles are those conditions, situations, and so 
forth that impede change and that are perceived to be real by 
those in the vocational education setting. If obstacles are not 
perceived as problems by those in the setting, then attempts to 
overcome those obstacles become solo efforts by the "evaluator" 
as expert. As a result, participants in the setting become 
removed from the evaluation effort, and the power of critical 
evaluation is forfeited. 

Transformative actions are attempts to penetrvite and remove 
obstacles to program improvement? that is, they are efforts to 
create change. Purchasing new or updated equipment for the 
school's vocational shops may be an example of a transformative 
action. Effective transformative actions solve problems and may 
cause new interpretive explanations of reality to emerge as wel' . 
The school that purchases new equipment may discover th^t another 
problem is created because teachers cannot operate the new 
equipment. Thus, new interpretations of the situation may 
illuminate more problems and, consequently, call for further 
actions. In essence, this reality transformation, in which 
actions lead to the perception of new problems and their 
resolution, is a perpetual process. 



Stakeholder Involvement in Critical Evaluation 

The largest asset of a vocational education program is human 
beings. Those individuals involved in and affected by the pro- 
gram (stakeholders) — students, teachers, administrators, counsel- 
ors, parents, employers, and others — are the key ingredients in 
the vocational education program's success or failure. Substan- 
tial evidence .suggests that successful businesses have adopted 
this principle and have moved from "trusting the system" to 
"trusting people." Those businesses have learned that worker 
output is increased when individuals are given a chance to influ- 
ence their own destiny. 

Successful schools have also discovered that trust in those 
people Who are stakeholders results in more productive schools. 
This principle is not, however, universally practiced. Many of 
the suggested solutions to the problems of vocational education 
cannot be imposed at the State or local management level. These 



"quick- fix" solutions do not involve stakeholders, as is neces- 
sary for effective program improvement. 

Currently, much of the change in vocational education occurs 
m a very disruptive fashion, frequently in reaction to some 
external pressure group or legislative initiative. A major 
strength of the ongoing involvement of stakeholders in the criti- 
cal evaluation process is that change is not disruptive, but 
evolves in keeping with the changing needs of society and of the 
vocational education reality. Critical evaluation in vocational 
education focuses on program improvement that results from the 
intensive participation of stakeholders in identifying the prob- 
lan, collecting infomation about the problem, and designing 
strategies for solving the problem. 



Evaluation as a Continuing Educative Process 

A fundamental difference between critical evaluation and 
most other forms, of evaluation currently practiced in vocational 
education is the involvement of stakeholders. This involvement 
is not to be construed as "window dressing." The spirit of in- 
novation and experimentation so critical to program improvement 
is dependent upon sustained and rigorous dialogue among stake- 
holders. A major aim is for stakeholders to, better understand 
their positions and the culture in which they operate. Enlight- 
ened stakeholders are in a much bf^tter position to alter those 
conditions they find detrimental co excellent programs. 

The educative process is a continuing activity. Stake- 
holders viewing their past actions can see how social conditions 
and remedies suggested for problems may interact both to improve 
progratus and contribute to new problem areas. Continuing in- 
volvement in the evaluation process helps keep stakeholders aware 
of changing conditions and encourages the development of under- 
standings and actions appropriate for program improvement. 
Through this process, critical evaluation permits stakeholders to 
attain a deeper awareness of the reality that shapes the voca- 
tional education program and of their capacity to improve that 
reality. 



Critical Evaluation at Different Levels 

Because the extensive involvement of stakeholders is an es- 
sential part of critical evaluation, it is most suited for use at 
the local education level. This is consistent with the fact that 
school improvement is ^ basically a local initiative. 

However, some elements of critical evaluation are applicable 
at every level. Certainly, the need to involve stakeholders in 
State and Federal evaluation efforts is clear. The advantages of 
having stakeholders engage in critical reflection and discourse 
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are not confined to the local level. However, State and federal 
demands for program accountability often results in a reliance on 
quantitative data. Attempts to develop more qualitative under- 
standings of the program are often de-emphasized . Nevertheless, 
the latter is essential for local school program improvement. 



Distinguishing Features 

We have suggested that critical evaluation can serve as an 
alternative to conventional ways of evaluating vocational educa- 
tion prpgrams. What, then, distinguishes critical evaluation 
from current evaluation efforts? 



Understanding the Past 

Vocational education programs do not exist outside of their 
historical context. A historical understanding of the ways the 
vocational education program is based on the community and school 
culture and on the experiences of stakeholders is an essential 
aspect of critical evaluation. Through reflection and dis- 
course, stakeholders can formulate explanations of the program^ s 
development. By understanding the past, stakeholders will be in 
a better position to propose changes for program improvement . 



Communication 

As has been emphasized, discourse among stakeholders is a 
key feature of critical evaluation. These intense discussions 
among stakeholders involve self-reflection and are intended to 
reveal unexamined motives, meanings, and intentions. 



Stakeholder Involvement 

Critical evaluation relies on and promotes the democratic 
participation of stakeholders. Vocational education is a partic- 
ularized program that reflects the interests, motivations, and 
needs of a complex and diverse group of stakeholders. Many of 
the benefits of extensive and in-depth stakeholder involvement 
are long-range. Because of their involvement, fundamental 
changes in stakeholders* attitudes can result in significant 
program changes. n 

Continuing Evaluation Effort 

Critical evaluation does not have discrete beginning and 
ending points. It is a continuous process in v^ich problem 
clarification, problem interpretation, and problem solving always 
interact. As the context changes, the problem changes, the 




information needed to examine the problem changes, and the most 
feasible solution may change. The selection of a solution may 
create conditions that vdll contribute to other problems. Pro- 
gram improvement is not a static affair. It demands attention 
through an interactive, continuing evaluation process. 
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educative function); however, those discussed above are the most 
important ones. A careful study of the features that distinguish 
critical evaluation from current evaluation efforts in vocational 
education reveals the enormous potential contribution critical 
evaluation can make to program improvement. 

considerations for Implementation 

• Time 

Vocational educators considering the implementation of 
critical evaluation should recognize that a substantial amount of 
time will be required for stakeholders to engage in reflection 
and discourse. There is no magic answer to the question, "How 
much time will be required?" Time requirements depend on the 
cotiplexity and diversity of the program, the nature of the prob- 
lems studied, and so forth. The time required should be viewed 
as a solid investment in improved programs. 

Democratic Participation 

For critical evaluation to achieve optimal results, it is 
essential that school administrators create an open and noncoer- 
cive environment, giving access to information and encouraging 
discussion of all aspects of evaluation problems. Stakeholders 
must be convinced that any canments they make will not be held 
against them. Participants in the evaluation process also need 
to understand how their reconmendations will affect decision 
making and policymaking. 

Truth 

Critical evaluation is committed to seeking valid conclu- 
sions based on historical perspectives and an understanding of 
underlying beliefs, values, and meanings. This cctnmitment is 
considerably different than that of current evaluation efforts. 
Those deciding to initiate critical evaluation should recognize 
that vfriat vocational education evaluation approaches have thought 
of as "truth" may be substantially i^svised or altered in critical 
evaluation. 
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CHAPTER 3 



ANTICIPATED BENEFITS OP USING CRITICAL EVALUATION 



Research on effective schools suppoirts the fact that impact 
on learner achievement is possible. The challenge then is to 
implement a process that will enable schools to improve their 
vocational education programs. 

The research and literature regarding effective schools and 
the change process strongly suggest that program improvement is 
much more likely to occur when there is-- 

o involvement of the stalceholders in the program 
improvement process; 

o attention to the complex interrelationships of 
values, beliefs, attitudes, and norms that exist 
among stakeholders; 

o an understanding of the historical aspects of a 

program's development and of the current school and 
ccxnmunity culture in v^ich the program is situated; 

o an atmosphere among stakeholders that encourages 

nonthreatening intellectual discussion that can lead 
to consensus and unity. 

The critical evaluation framework proposed for vocational 
education directly addresses these key considerations for imple- 
menting school improvement efforts. Following are some of the 
major benefits to be expected when using critical evaluation. 



Stakeholder Support 

It is no secret that individuals are more enthusiastic in 
their support of ideas and changes vghen they have been meaning- 
fully involved in the generation of the ideas and changes. The 
proposed critical evaluation framework relies on extensive in- 
volvement of stakeholders as an essential ingredient of the eval- 
uation effort, involvement at a much greater depth than that . 
typical of most current evaluation effbrts. Without question, 
this kind of involvement will result in stakeholders having 
intense feelings and beliefs about the value of their proposals 
for program improvement. 

Meaningful Information 

Information about a program fails to achieve optimal impact 
unless it is interpreted in light of stakeholder values, beliefs. 
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and feelings. The critical evaluation framework permits the use \ 
of quantitative and qualitative information appropriate to the 
problem under evaluation. This information becomes more meaninq**'^'=' 
ful as stakeholders interact vdth each other and interpret the 
information in light of their feelings, beliefs, and attitudes. 

Context-specific Recommendations 

When the critical evaluation framework is used,, the actions 
identified as appropriate for improving the program will be spe- 
cific to a particular situation. The emphasis on the program* s 
historical development and the interaction among program stake- 
holders will help assure that reccmmendations for program im- 
provement are tailored to meet the needs of the specific setting. 
This should enhance the utility of evaluation findings 
considerably. ^ 

Consensus on Recommendation^ 

Most current evaluation efforts are dominated by an individ- 
ual or small group of individuals, representative of some central 
authority figure. Such ef'forts are certainly not stakeholder 
based. In critical evaluation, honest and earnest attempts are 
made to achieve conseinsus among stakeholders. This consensus 
building in all phases of critical evaluation gives recommenda- 
tions • for program improvement a better chance of being 
implemented. 

Critical evaluation is a fundamental and radical departure 
from current vocational education program evaluation efforts. It 
incorporates both the common sense of stakeholders and a strong 
theoretical base. Vocational educators searching for a more 
flexible, sensible, and practical approach to program improvement 
should explore thoroughly the merits of critical evaluation. 
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